Adaptive bimodal sensor fusion for automatic speechreading

We present recent work on improving the performance of automated speech recognizers by using additional visual in formation Lip Speechreading achieving error reduction of up to This paper focuses on di erent methods of combining the visual and acoustic data to improve the recognition performance We show this on an extension of an existing state of the art speech recognition system a modular MS TDNN We have developed adaptive combi nation methods at several levels of the recognition network Additional information such as estimated signal to noise ra tio SNR is used in some cases The results of the di er ent combination methods are shown for clean speech and data with arti cial noise white music motor The new combination methods adapt automatically to varying noise conditions making hand tuned parameters unnecessary